Systems and methods for video-based monitoring of vital signs

ABSTRACT

The present invention relates to the field of medical monitoring, and in particular non-contact, video-based monitoring of pulse rate, respiration rate, motion, and oxygen saturation. Systems and methods are described for capturing images of a patient, producing intensity signals from the images, filtering those signals to focus on a physiologic component, and measuring a vital sign from the filtered signals.

RELATED APPLICATIONS

This application claims priority to and the benefit of U.S. ProvisionalApplication No. 62/297,682 (filed Feb. 19, 2016); U.S. ProvisionalApplication No. 62/335,862 (filed May 13, 2016); and U.S. ProvisionalApplication No. 62/399,741 (filed Sep. 26, 2016), the contents of whichare incorporated herein by reference in their entirety.

BACKGROUND

Many conventional medical monitors require attachment of a sensor to apatient in order to detect physiologic signals from the patient andtransmit detected signals through a cable to the monitor. These monitorsprocess the received signals and determine vital signs such as thepatient's pulse rate, respiration rate, and arterial oxygen saturation.An example of a prior art monitoring system 100 is shown in FIG. 1. Thesystem 100 includes a monitor 110 and a sensor 112 connected to themonitor 110 by a cable 114. In the example of FIG. 1, the monitor 110 isa pulse oximeter, and the sensor 112 is a finger sensor including twolight emitters and a photodetector. The sensor 112 emits light into thepatient's finger, detects light transmitted through the patient'sfinger, and transmits the detected light signal through the cable 114 tothe monitor 110. The monitor 110 includes a processor that processes thesignal, determines vital signs (including pulse rate, respiration rate,and arterial oxygen saturation), and displays them on an integrateddisplay 116.

Other monitoring systems include other types of monitors and sensors,such as electroencephalogram (EEG) sensors, blood pressure cuffs,temperature probes, and others.

Many of these conventional monitors require some type of cable or wire,such as cable 114 in FIG. 1, physically connecting the patient to themonitor. As a result, the patient is effectively tethered to themonitor, which can limit the patient's movement around a hospital room,restrict even simple activities such as writing or eating, and preventeasy transfer of the patient to different locations in the hospitalwithout either disconnecting and connecting new monitors, or moving themonitor with the patient.

Some wireless, wearable sensors have been developed, such as wirelessEEG patches and wireless pulse oximetry sensors. Although these sensorsimprove patient mobility, they introduce new problems such as batteryconsumption, infection risk from re-use on sequential patients, highcost, and bulky designs that detract from patient compliance andcomfort.

Video-based monitoring is a new field of patient monitoring that uses aremote video camera to detect physical attributes of the patient. Thistype of monitoring may also be called “non-contact” monitoring inreference to the remote video sensor, which does not contact thepatient. The remainder of this disclosure offers solutions andimprovements in this new field.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view of a pulse oximetry monitor and sensoraccording to the prior art.

FIG. 2A is schematic view of a video-based patient monitoring systemaccording to an embodiment of the invention.

FIG. 2B is schematic view of a video-based patient monitoring systemmonitoring multiple patients according to an embodiment of theinvention.

FIG. 3A depicts an image frame from a video signal according to anembodiment of the invention.

FIG. 3B depicts an image frame from a video signal according to anembodiment of the invention.

FIG. 4A depicts light intensity signals from the video signal of FIG.3A.

FIG. 4B depicts light intensity signals from the video signal of FIG.4A.

FIG. 5A depicts an image frame according to an embodiment of theinvention.

FIG. 5B depicts an image frame according to an embodiment of theinvention.

FIG. 5C is a chart of a light intensity signal from a first region ofinterest according to an embodiment of the invention.

FIG. 5D is a chart of a light intensity signal from a second region ofinterest according to an embodiment of the invention.

FIG. 5E is a flowchart of a method for measuring a vital sign from acombined region of interest according to an embodiment of the invention.

FIG. 5F is a flowchart of a method for dynamically updating anddisplaying a color signal from a moving region of interest according toan embodiment of the invention.

FIG. 6A is a flowchart of a method of determining vital signs from avideo signal according to an embodiment of the invention.

FIG. 6B is a chart of contact-oximeter-based and video-based vital signs(heart rate and SpO2) over time according to an embodiment of theinvention.

FIG. 7 is a flowchart of a method of calibrating video-based pulseoximetry according to an embodiment of the invention.

FIG. 8 is a chart of video-based and contact-based measurements ofarterial oxygen saturation over time, including a desaturation event,according to an embodiment of the invention.

FIG. 9 is a flowchart of a method for calibrating a video cameraaccording to an embodiment of the invention.

FIG. 10A is a chart of red, green, and blue pixel signals over time anda corresponding frequency transform according to an embodiment of theinvention.

FIG. 10B is a flowchart of a method of calculating heart rate from avideo signal utilizing a frequency accumulator, according to anembodiment of the invention. FIG. 10B-1 is a continuation page for FIG.10B.

FIG. 11 depicts a patient in an image frame according to an embodimentof the invention.

FIG. 12 is a bottom view of a calibration strip according to anembodiment of the invention.

FIG. 13 is a set of charts showing three source signals and threeindependent component signals according to an embodiment of theinvention.

FIG. 14 is a chart of contact-oximeter-based and video-based ICA-derivedheart rate values over time according to an embodiment of the invention.

FIG. 15 is a schematic chart illustrating an independent componentsignal and two source signals, according to an embodiment of theinvention.

FIG. 16 is a flowchart illustrating a method for utilizing anindependent component signal, according to an embodiment of theinvention.

FIG. 17A is a flowchart illustrating a method for identifyingnon-physiologic frequencies according to an embodiment of the invention.

FIG. 17B is a flowchart illustrating a method for identifyingnon-physiologic frequencies according to an embodiment of the invention.

FIG. 18 is a schematic cut-away view of an optical splitter according toan embodiment of the invention.

FIG. 19 is a scatter plot of video-calculated heart rate measurementsagainst reference heart rate measurements, according to an embodiment ofthe invention.

FIG. 20 is a scatter plot of video-based respiration rate measurementsagainst reference respiratory rate measurements, according to anembodiment of the invention.

FIG. 21 is a scatter plot of video-based SpO2 measurements againstreference SpO2 measurements, according to an embodiment of theinvention.

SUMMARY

A calibration strip may be used as explained in more detail below. In anembodiment, a video-based method of measuring a patient's vital signincludes providing a calibration strip comprising a substrate with firstand second opposite surfaces, an adhesive on the first surface of thesubstrate for adhering to a patient, and a visible scale on the secondsurface for viewing by a video camera; detecting, by the video camera, afirst light signal from the scale and a second light signal from thepatient, within the same field of view; adjusting a calibration of thevideo camera based on a measurement of the first light signal; applyingthe calibration to the second light signal; measuring a vital sign ofthe patient from the calibrated second light signal; and outputting themeasured vital sign for further processing or display.

In an embodiment, the scale comprises a greyscale, and the measurementof the first light signal comprises a measurement of a first intensityof at least a portion of the greyscale. In an embodiment, the methodincludes, at a later time, measuring a second intensity that differsfrom the first intensity by an amount, and further adjusting thecalibration based on the measured second intensity. In an embodiment,the method also includes further adjusting the calibration based on thesecond intensity comprises adjusting a coefficient in proportion to theamount. In an embodiment, the method includes determining that theamount exceeds a threshold, prior to adjusting the coefficient. In anembodiment, detecting the second light signal comprises combining lightfrom two or more non-contiguous regions of exposed skin of the patient.

In an embodiment, the scale comprises a color map comprising a pluralityof colors, and the measurement of the first light signal comprises ameasurement of a color value of one of the plurality of colors. In anembodiment, adjusting the calibration comprises comparing the colorvalue to a reference color value and identifying a difference. In anembodiment, the reference color value comprises a baseline color valuemeasured from the first light signal at a first time. In an embodiment,adjusting the calibration comprises determining that the differenceexceeds a threshold, and adjusting a coefficient based on thedifference.

In an embodiment, the scale comprises a greyscale, and wherein themeasurement of the first light signal comprises a measurement of a whitevalue of the greyscale. In an embodiment, applying the calibrationcomprises white balancing the first and second light signals.

In an embodiment, the method includes operating a second video camera tomonitor a second patient, and adjusting a calibration of the secondvideo camera to match the adjusted calibration of the first videocamera.

In an embodiment, the method includes detecting motion based on movementof the scale within the field of view, and generating a motion signalbased on the detected motion.

In an embodiment, the method includes measuring oxygen saturation fromthe calibrated second light signal. The calibrated second light signalcomprises two of a red signal, a green signal, and a blue signal, andmeasuring the oxygen saturation comprises measuring a ratio of the twoof the red, green, and blue signals. In an embodiment, detecting thesecond light signal comprises combining light from two or morenon-contiguous regions of exposed skin of the patient.

In an embodiment, the video comprises an optical splitter, and thecalibrated second light signal comprises two light signals output fromthe optical splitter.

In an embodiment, a system for video-based measurement of a patient'spulse rate includes a video camera positioned remote from a patient, thevideo camera having a field of view encompassing exposed skin of thepatient; a calibration strip positioned within the field of view, thecalibration strip comprising a scale viewable by the camera; and ahardware memory coupled to the video camera by wired or wirelesscommunication, the memory storing instructions for instructing aprocessor to: detect a first light intensity signal from the scale and asecond light intensity signal from the exposed skin of the patient;adjust a calibration of the video camera based on a measurement of thefirst light intensity signal; apply the calibration to the second lightintensity signal; measure a pulse rate of the patient from thecalibrated second light intensity signal; and output the measured pulserate for further processing or display. In an embodiment, thecalibration strip comprises first and second light emitters. In anembodiment, the calibration strip is devoid of a photodetector.

Independent component analysis may be used, as explained in more detailbelow. In an embodiment, a method for measuring blood oxygen saturationof a patient includes receiving, from a video camera, a video signalencompassing exposed skin of a patient; extracting from the video signaltime-varying red, green, and blue signals; decomposing the red, green,and blue signals into a component signal having a primary frequency at apulse rate of the patient; identifying, in the component signal, anindividual pulse representative of a heart beat; locating acorresponding portion of two of the red, green, and blue signals; andmeasuring blood oxygen saturation of the patient from the locatedcorresponding portions of the two signals.

In an embodiment, the method includes determining and displaying a pulserate measured from the primary frequency of the component signal. In anembodiment, an audio beep is triggered in synchrony with the locatedcorresponding portion of one of the two signals or in synchrony with theidentified individual pulse in the component signal. In an embodiment,the located portions of the two signals comprise cardiac pulses, and,for each of the two signals, the located cardiac pulse is added to aweighted average pulse. In an embodiment, measuring blood oxygensaturation comprises calculating a ratio of ratios of the weightedaverage pulses of the two signals. In an embodiment, extracting the red,green, and blue signals comprises selecting pixels within the imageframe that exhibit a modulation at the primary frequency. In anembodiment, the selected pixels are non-contiguous.

In an embodiment, extracting the red, green, and blue signals comprisesselecting pixels within the image frame exhibiting a modulation that isat the primary frequency and that has an amplitude above a threshold.

In an embodiment, a method for measuring a pulse rate of a patientincludes receiving, from a video camera, a video signal having a fieldof view encompassing exposed skin of a patient; identifying, within thevideo signal, regions of pixels that exhibit a modulation above anamplitude threshold; extracting from the identified regions time-varyingred, green, and blue signals; decomposing the red, green, and bluesignals into a component signal having a primary frequency at a pulserate of the patient; measuring the pulse rate from the primary frequencyof the component signal; and outputting the measured pulse rate forfurther processing or display.

In an embodiment, the method also includes identifying, in the componentsignal, individual pulses representative of individual heart beats; foreach identified pulse, locating a corresponding portion of two of thered, green, and blue signals; and measuring blood oxygen saturation ofthe patient from the located corresponding portions of the two signals.

A frequency accumulator may be used, as explained in more detail below.In an embodiment, a method for video-based monitoring of a patient'spulse rate includes generating a video signal from a video camera havinga field of view exposed to a patient, the video signal comprising atime-varying intensity signal for each of a plurality of pixels in thefield of view; combining the intensity signals within a region of thefield of view to produce a regional intensity signal; transforming theregional intensity signal into the frequency domain to produce aregional frequency signal; over a sliding time window, identifying peaksin the regional frequency signal; over a period of time, accumulatingthe identified peaks; selecting a median frequency from the accumulatedpeaks; updating a running average pulse rate of a patient, whereinupdating comprises: converting the median frequency into a measuredpulse rate; and adding the measured pulse rate to the running average toproduce an updated average pulse rate; and outputting the updatedaverage pulse rate for display.

In an embodiment, the period of time is one second. In an embodiment,identified peaks from the accumulated peaks are removed based on an ageof the identified peaks. In an embodiment, the method includes, atrepeated intervals, discarding the accumulated peaks and repeating thesteps of identifying peaks, accumulating peaks, selecting the medianfrequency, updating the running average pulse rate, and outputting theupdated average pulse rate.

In an embodiment, adding the measured pulse rate to the running averagecomprises applying a weight to the measured pulse rate based on aquality of the regional frequency signal. In an embodiment, the qualityof the regional frequency signal is measured by a variability of theaccumulated peaks over the period of time. In an embodiment, the qualityof the regional frequency signal is measured by an amplitude of theaccumulated peaks. In an embodiment, the quality of the regionalfrequency signal is measured by a signal to noise ratio of the regionalfrequency signal.

In an embodiment, frequency peaks outside of a physiologic limit arediscarded. In an embodiment, the measured pulse rate is discarded whenit differs from the average pulse rate by more than a defined amount.

In an embodiment, the method includes updating an average respirationrate of the patient, wherein updating the average respiration ratecomprises: selecting a second median frequency from the identifiedpeaks; converting the second median frequency into a measuredrespiration rate; and adding the measured respiration rate to theaverage respiration rate to produce an updated average respiration rate;and outputting the updated average respiration rate for display.

In an embodiment, selecting the region of the field of view is based ona strength of modulations of the pixels in the region. In an embodiment,the region comprises two or more non-adjacent groups of pixels.

The frame rate may be adjusted to reject noise, as explained in moredetail below. In an embodiment, a method for video-based monitoring of avital sign of a patient includes receiving a video signal from a videocamera having a field of view exposed to a patient, the video signalcomprising a time-varying intensity signal for each of a plurality ofpixels in the field of view; combining the intensity signals of selectedpixels to produce a time-varying regional intensity signal; transformingthe regional intensity signal into the frequency domain to produce aregional frequency signal; operating the video camera at a first framerate during a first period of time, and at a second, different framerate during a second, subsequent period of time; identifying, in theregional frequency signal, a noise peak at a first frequency during thefirst period of time that moves to a second, different frequency upon atransition from the first period of time to the second period of time;filtering the regional intensity signal to remove the frequency of thenoise peak; and measuring a vital sign of the patient from the filteredregional intensity signal.

In an embodiment, the method also includes identifying a stationary peakthat remains stationary in the frequency domain upon the transition fromthe first period of time to the second period of time, and measuring thevital sign from the identified stationary peak. In an embodiment, thevital sign comprises pulse rate, and measuring the vital sign comprisesconverting the frequency of the identified stationary peak into thepulse rate. In an embodiment, combining the intensity signals ofselected pixels comprises selecting the pixels that exhibit modulationsat a shared frequency. In an embodiment, the selected pixels arenon-contiguous.

In an embodiment, a method for video-based monitoring of a vital sign ofa patient includes receiving a video signal from a video camera having afield of view exposed to a patient, the video signal comprising atime-varying intensity signal for each of a plurality of pixels in thefield of view; combining the intensity signals of selected pixels toproduce a time-varying regional intensity signal; transforming theregional intensity signal into the frequency domain to produce aregional frequency signal; operating the video camera at a first framerate during a first period of time, and at a second, different framerate during a second, subsequent period of time; identifying, in theregional frequency signal, a stationary peak that is stationary upon atransition from the first period of time to the second period of time;and measuring a vital sign of the patient from the identified stationarypeak.

In an embodiment, the method also includes identifying, in the regionalfrequency signal, a noise peak that is non-stationary upon thetransition from the first period of time to the second period of time,and filtering the regional intensity signal to remove the noise peak.

In an embodiment, a method for video-based monitoring of a vital sign ofa patient includes receiving a video signal from a video camera having afield of view exposed to a patient, the video signal comprising atime-varying intensity signal for each of a plurality of pixels in thefield of view; operating the video camera at a frame rate that changesaccording to a change trajectory over a period of time; combining theintensity signals of selected pixels to produce a combined intensitysignal; transforming the combined intensity signal into the frequencydomain to produce a frequency signal; identifying, in the frequencysignal, a noise peak that moves in synchrony with the change trajectory;filtering the combined intensity signal to remove the noise peak; andmeasuring a vital sign of the patient from the filtered intensitysignal.

In an embodiment, the method also includes identifying, in the frequencysignal, a physiologic peak that is stationary over the period of time,and measuring the vital sign from the physiologic peak. In anembodiment, the physiologic peak corresponds to a physiologic frequencyof the patient. In an embodiment, the physiologic frequency is pulserate. In an embodiment, the physiologic frequency has a period that issmaller than the period of time.

In an embodiment, the change trajectory comprises sweeping the framerate at a constant sweep rate over the period of time. In an embodiment,identifying the noise peak comprises identifying a peak that moves atthe sweep rate. In an embodiment, the change trajectory comprises threeor more different, discrete frame rates. In an embodiment, the framerate is fixed after the noise peak has been identified.

In an embodiment, a method for video-based monitoring of a vital sign ofa patient includes receiving a video signal from a video camera having afield of view exposed to a patient, the video signal comprising atime-varying intensity signal for each of a plurality of pixels in thefield of view; combining the intensity signals of selected pixels toproduce a combined intensity signal; transforming the combined intensitysignal into the frequency domain over first and second different timewindows to produce first and second frequency transforms; identifying,in the first and second frequency transforms, a noise peak that remainsstationary in the first and second frequency transforms; filtering thecombined intensity signal to remove the noise peak; and measuring avital sign of the patient from the filtered intensity signal.

A region of interest may be displayed to a user, as explained in moredetail below. In an embodiment, a video-based method for measuring avital sign of a patient includes receiving a video signal from a videocamera having a field of view exposed to a patient; displaying on adisplay screen a portion of the video signal; receiving a first userinput from a user, the first user input identifying a location of afirst region of interest within the video signal; extracting from thefirst region of interest a first intensity signal comprising atime-varying intensity of one or more pixels in the first region;measuring a first vital sign of the patient from the first intensitysignal; displaying on the display screen a time-varying modulation ofthe first intensity signal; receiving a second user input from a user,the second user input indicating that the location has been moved to asecond, different region of interest; extracting from the second regionof interest a second intensity signal comprising a time-varyingintensity of one or more pixels in the second region; measuring a secondvital sign of the patient from the second intensity signal; anddisplaying on the display screen a time-varying modulation of the secondcolor signal.

In an embodiment, the first vital sign comprises heart rate and thesecond vital sign comprises respiration rate. In an embodiment, themodulations of the first and second color signals are displayed on thedisplay screen simultaneously. In an embodiment, the second user inputfurther comprises a path from the first region of interest to the secondregion of interest, and wherein the method further comprises identifyinga plurality of intermediate regions of interest along the path,extracting an intermediate intensity signal from one of the intermediateregions, and displaying on the display screen a modulation of theintermediate intensity signal. In an embodiment, the first region ofinterest comprises two or more non-adjacent groups of pixels.

In an embodiment, a video-based method for monitoring a patient includesdisplaying on a display screen a video signal from a video camera havinga field of view exposed to a patient; receiving a first user input froma user, the first user input identifying a location of a region ofinterest within the video signal; extracting from the region of interesta first intensity signal comprising a time-varying intensity of one ormore pixels in the region of interest; displaying on the display screenthe first intensity signal; receiving from the user a second user inputthat moves the region of interest along a path; continually updating thelocation of the region of interest along the path; continually updatingthe intensity signal extracted from the moving region of interest; anddisplaying on the display screen the continually updated intensitysignal.

In an embodiment, the method also includes identifying a modulation ofthe intensity signal, and measuring a physiologic rate of the patientfrom the modulation. In an embodiment, the physiologic rate is pulserate. In an embodiment, a transform of the intensity signal into thefrequency domain is displayed.

DETAILED DESCRIPTION

The present invention relates to the field of medical monitoring, and inparticular non-contact, video-based monitoring of pulse rate,respiration rate, motion, activity, and oxygen saturation. Systems andmethods are described for receiving a video signal in view of a patient,identifying a physiologically relevant area within the video image (suchas a patient's forehead or chest), extracting a light intensity signalfrom the relevant area, filtering those signals to focus on aphysiologic component, and measuring a vital sign from the filteredsignals. The video signal is detected by a camera that views but doesnot contact the patient. With appropriate selection and filtering of thevideo signal detected by the camera, the physiologic contribution to thedetected signal can be isolated and measured, producing a useful vitalsign measurement without placing a detector in physical contact with thepatient. This approach has the potential to improve patient mobility andcomfort, along with many other potential advantages discussed below.

As used herein, the term “non-contact” refers to monitors whosemeasuring device (such as a detector) is not in physical contact withthe patient. Examples include cameras, accelerometers mounted on apatient bed without contacting the patient, radar systems viewing thepatient, and others. “Video-based” monitoring is a sub-set ofnon-contact monitoring, employing one or more cameras as the measuringdevice. In an embodiment, the camera produces an image stack, which is atime-based sequence of images of the camera's field of view. The cameramay be considered a “video” camera if the frame rate is fast enough tocreate a moving, temporal image signal.

Remote sensing of a patient in a video-based monitoring system presentsseveral new challenges. One challenge is presented by motion. Theproblem can be illustrated with the example of pulse oximetry.Conventional pulse oximetry sensors include two light emitters and aphotodetector. The sensor is placed in contact with the patient, such asby clipping or adhering the sensor around a finger, toe, or ear of apatient. The sensor's emitters emit light of two particular wavelengthsinto the patient's tissue, and the photodetector detects the light afterit is reflected or transmitted through the tissue. The detected lightsignal, called a photoplethysmogram (PPG), modulates with the patient'sheartbeat, as each arterial pulse passes through the monitored tissueand affects the amount of light absorbed or scattered. Movement of thepatient can interfere with this contact-based oximetry, introducingnoise into the PPG signal due to compression of the monitored tissue,disrupted coupling of the sensor to the finger, pooling or movement ofblood, exposure to ambient light, and other factors. Modern pulseoximeters employ filtering algorithms to remove noise introduced bymotion and to continue to monitor the pulsatile arterial signal.

However, movement in non-contact pulse oximetry creates differentcomplications, due to the extent of movement possible between thepatient and the camera, which acts as the detector. Because the camerais remote from the patient, the patient may move toward or away from thecamera, creating a moving frame of reference, or may rotate with respectto the camera, effectively morphing the region that is being monitored.Thus, the monitored tissue can change morphology within the image frameover time. This freedom of motion of the monitored tissue with respectto the detector introduces new types of motion noise into thevideo-based signals.

Another challenge is the contribution of ambient light. In this context,“ambient light” means surrounding light not emitted by components of themedical monitor. In contact-based pulse oximetry, the desired lightsignal is the reflected and/or transmitted light from the light emitterson the sensor, and ambient light is entirely noise. The ambient lightcan be filtered, removed, or avoided in order to focus on the desiredsignal. In contact-based pulse oximetry, contact-based sensors can bemechanically shielded from ambient light, and direct contact between thesensor and the patient also blocks much of the ambient light fromreaching the detector. By contrast, in non-contact pulse oximetry, thedesired physiologic signal is generated or carried by the ambient lightsource; thus, the ambient light cannot be entirely filtered, removed, oravoided as noise. Changes in lighting within the room, includingoverhead lighting, sunlight, television screens, variations in reflectedlight, and passing shadows from moving objects all contribute to thelight signal that reaches the camera. Even subtle motions outside thefield of view of the camera can reflect light onto the patient beingmonitored. Thus new filtering techniques are needed to isolate thephysiologic signal from this combined ambient light signal.

If these challenges are addressed, non-contact monitoring such asvideo-based monitoring can deliver significant benefits. Somevideo-based monitoring can reduce cost and waste by reducing usage ofdisposable contact sensors, replacing them with reusable camera systems.Video monitoring may also reduce the spread of infection, by reducingphysical contact between caregivers and patients (otherwise incurredwhen the caregiver places, adjusts, or removes the contact sensor on thepatient). Some remote video cameras may improve patient mobility andcomfort, by freeing patients from wired tethers or bulky wearablesensors. This untethering may benefit patients who need exercise andmovement. In some cases, these systems can also save time forcaregivers, who no longer need to reposition, clean, inspect, or replacecontact sensors. Another benefit comes from the lack of sensor-offalarms or disruptions. A traditional contact-based system can lose thephysiologic signal when the contact sensor moves or shifts on thepatient, triggering alarms that are not actually due to a change inphysiology. In an embodiment, a video-based system does not dropreadings due to sensors moving or falling off the patient (sensor-off)or becoming disconnected from the monitor (sensor-disconnect), and thuscan reduce nuisance alarms. In an embodiment, a video-based monitor,such as a pulse oximeter, operates without sensor-off orsensor-disconnect alarms. For example, a video-based monitor can triggeran alarm based on stored alarm conditions, where the stored alarmconditions omit a sensor-off or sensor-disconnect alarm.

Various embodiments of the present invention are described below, toaddress some of these challenges. FIG. 2A shows a video-based remotemonitoring system 200 and a patient 212, according to an embodiment. Thesystem 200 includes a non-contact detector 210 placed remote from thepatient 212. In this embodiment, the detector 210 includes a camera 214,such as a video camera. The camera 214 is remote from the patient, inthat it is spaced apart from and does not contact the patient. Thecamera includes a detector exposed to a field of view 216 thatencompasses at least a portion of the patient 212. In some embodiments,the field of view 216 encompasses exposed skin of the patient, in orderto detect physiologic signals visible from the skin such as arterialoxygen saturation (SpO2 or SvidO2). The camera generates a sequence ofimages over time. A measure of the amount, color, or brightness of lightwithin all or a portion of the image over time is referred to as a lightintensity signal. In an embodiment, each image includes atwo-dimensional array or grid of pixels, and each pixel includes threecolor components—for example, red, green, and blue. A measure of one ormore color components of one or more pixels over time is referred to asa “pixel signal,” which is a type of light intensity signal. The cameraoperates at a frame rate, which is the number of image frames taken persecond (or other time period). Example frame rates include 20, 30, 40,50, or 60 frames per second, greater than 60 frames per second, or othervalues between those. Frame rates of 20-30 frames per second produceuseful signals, though frame rates above 50 or 60 frames per second arehelpful in avoiding aliasing with light flicker (for artificial lightshaving frequencies around 50 or 60 Hz).

The detected images are sent to a monitor 224, which may be integratedwith the camera 214 or separate from it and coupled via wired orwireless communication with the camera (such as wireless communication220 shown in FIG. 2A). The monitor 224 includes a processor 218, adisplay 222, and hardware memory 226 for storing software and computerinstructions. Sequential image frames of the patient are recorded by thevideo camera 214 and sent to the processor 218 for analysis. The display222 may be remote from the monitor 224, such as a video screenpositioned separately from the processor and memory.

FIG. 2B shows the system 200 being implemented to monitor multiplepatients, such as patients 212A and 212B. Because the detector 214 inthe system is non-contact, it can be used to monitor more than onepatient at the same time. A method for this implementation will bedescribed in further detail below.

Two example image frames 300A and 300B are shown in FIGS. 3A and 3B,respectively. In an embodiment, these image frames are recorded by thesystem 200. Each image frame includes a patient's head 312 and uppertorso 310 in the field of view. The processor has identified a headregion 314 within each image frame 300A, 300B. The head region 314includes at least a portion of the patient's head, such as the face. Insome embodiments, the processor also infers a chest region 316, based onthe size and location of the head region 314 and empirical ratios ofhead and chest sizes and shapes. For example, from a rectangular faceregion of width w and height h, a forehead region may be inferred of asize 0.7*w and 0.3*h, centered horizontally and positioned with its topedge moved down from the top of the face region by a distance 0.25*h.From the same rectangular face region, a chest region may also beinferred at a size of 2*w and 0.75*h, centered horizontally andpositioned with its top edge below the bottom of the face region by adistance 0.25*h.

In an embodiment, the video camera records multiple sequential imageframes (such as image frames 300A and 300B) that each include the headregion 314 and chest region 316. The pixels or detected regions in thesesequential images exhibit subtle modulations caused by the patient'sphysiology, such as heartbeats and breaths. In particular, the colorcomponents of the pixels vary between the frames based on the patient'sphysiology. In one embodiment, the camera employs the Red/Green/Bluecolor space and records three values for each pixel in the image frame,one value each for the Red component of the pixel, the Blue component,and the Green component. Each pixel is recorded in memory as these threevalues, which may be integer numbers (typically ranging from 0 to 255for 8-bit color depth, or from 0 to 4095 for 12-bit color depth) orfractions (such as between 0 and 1). Thus, three one-dimensional vectorsfor each pixel in the field of view can be extracted from the videosignal.

These Red, Green, and Blue values change over time due to the patient'sphysiology, though the changes may be too subtle to be noticed by thenaked human eye viewing the video stream. For example, the patient'sheartbeat causes blood to pulse through the tissue under the skin, whichcauses the color of the skin to change slightly—causing the valuecorresponding to the Red, Green, or Blue component of each pixel to goup and down. These changes in the pixel signals can be extracted by theprocessor. The regions within the field of view where these changes arelargest can be identified and isolated to focus on the physiologicsignal. For example, in many patients, the forehead is well-perfusedwith arterial blood, so pixels within the patient's forehead exhibitheartbeat-induced modulations that can be measured to determine thepatient's heartrate.

To focus on this physiologic signal, the processor identifies a regionof interest (ROI) within the image frame. In an embodiment, the regionof interest includes exposed skin of the patient, such that thephysiologic properties of the skin can be observed and measured. Forexample, in the embodiment of FIG. 3A, one region of interest includes aforehead region 330, which includes part of the patient's forehead. Theprocessor determines the location of the patient's forehead within thehead region 314, for example based on empirical ratios for a human face,and divides the forehead into distinct regions, for example, regions 1A,2A, and 3A. In another embodiment, the region of interest does notinclude exposed skin. For example, in FIG. 3A, another region ofinterest includes the chest region 316 (which may be covered byclothing, bedding, or other materials on the patient). Pixels in thisregion may fluctuate with the patient's respiration rate, enabling thatrate to be measured even without viewing exposed skin of the patient.

Within an individual region of interest, the Red components of thepixels in that region are combined together to produce one time-varyingRed pixel signal from that region. The same is done for the Blue andGreen pixels. The result is three time-varying pixel signals from eachregion, and these are plotted in FIG. 4A. The plots in FIG. 4A arederived from the regions 1A, 2A, 3A, and 316 of FIG. 3A. FIG. 4A alsoshows a plot labeled “Combined Forehead.” The Combined Forehead plotshows the combined pixel signals from all three identified regions 1A,2A, and 3A, meaning that the Red components from all three regions arecombined together and plotted over time, as are the Green components andthe Blue components. Different sub-sets of regions can be combinedtogether to produce different combinations of pixel signals. Thoughthree forehead regions 1A, 2A, and 3A are shown in FIG. 3A, theforehead, or any other area of interest, can be sub-divided into more orfewer regions, in various shapes or configurations. (Other examples aredescribed in more detail below with reference to FIGS. 5A and 5B.) Pixelsignals can be combined by summing or averaging or weighted averaging.In an embodiment, the combined pixel signals are obtained by averagingthe Red (or Blue, or Green) color values of the pixels within theregion, so that regions of different sizes can be compared against eachother.

The pixels within a region may be combined together with a weightedaverage. For example, within a region, some pixels may exhibit strongermodulations than other pixels, and those stronger-modulating pixels canbe weighted more heavily in the combined pixel signal. A weight can beapplied to all of the pixels that are combined together, and the weightcan be based on quality metrics applied to the modulating intensitysignal of each pixel, such as the signal to noise ratio of the intensitysignal, a skew metric, an amplitude of a desired modulation (such asmodulations at the heart rate or respiration rate), or othermeasurements of the signal. Further, some pixels within the region maybe chosen to be added to the combined pixel signal for that region, andother pixels may be discarded. The chosen pixels need not be adjacent orconnected to each other; disparate pixels can be chosen and combinedtogether to create the resulting signal.

The plots in FIG. 4A show a clear pattern of repeating modulations orpulses over time. The pulses in each region 1A, 2A, 3A and in theCombined Forehead plot are caused by the patient's heart beats, whichmove blood through those regions in the patient's forehead, causing thepixels to change color with each beat. The heart rate of the patient canbe measured from these signals by measuring the frequency of themodulations. This measurement can be taken via a frequency transform ofthe signal (discussed below with reference to FIG. 10A and FIG. 4B) orvia a pulse recognition algorithm that identifies each pulse in thesignal (for example, by pulse size and shape, by zero crossings,maximums, or minimums in the derivative of the signal, and/or bychecking the skew of the derivative of the signal to identify a pulse asa cardiac pulse, which has a characteristically negative skew). Themodulations in the plot of the Chest region, in FIG. 4A, are caused bythe patient's breaths, which cause the chest to move in correspondencewith the breathing rate. The patient's breathing/respiration rate can bemeasured from this signal in the same way as just described for theheart rate (except for the skew approach). Respiration rate can beidentified from a region of the patient that moves with each breath,such as the chest, but need not include exposed skin.

FIG. 4B shows plots of the pixel streams from the corresponding regionsin FIG. 3B. However, in this case, the individual Red, Green, and Bluevalues within each region have been combined together, such as bysumming or averaging, to produce one time-varying signal from eachregion instead of three separate Red, Green, and Blue signals. Byviewing one combined signal from each region, the frequency of the heartrate or respiration rate may emerge more clearly. FIG. 4B also shows aFast Fourier Transform (FFT) in the Chest Region plot. The FFTidentifies the frequency content of the Chest signal, which reveals aprimary frequency peak and harmonics. The primary frequency peak is thepatient's respiration rate.

Though many embodiments herein are described with reference to pixelsand pixel values, this is just one example of a detected light intensitysignal. The light intensity signals that are detected, measured, oranalyzed may be collected from larger regions or areas, withoutdifferentiating down to groups of pixels or individual pixels. Lightsignals may be collected from regions or areas within an image, whetheror not such regions or areas are formed from pixels or mapped to aspatial grid. For example, time-varying light signals may be obtainedfrom any detector, such as a camera or light meter, that detects a unitof light measurement over time. Such units of light measurement may comefrom individual pixels, from groups or clusters of pixels, regions,sub-regions, or other areas within a field of view. It should also benoted that the term “pixel” includes larger pixels that are themselvesformed from aggregates, groups, or clusters of individual pixels.

In an embodiment, the Red, Green, and Blue values from the camera areconverted into different color spaces, and the color space that providesthe largest or most identifiable physiologic modulations is chosen. Inan embodiment, color values are converted into a combination of a colorvalue and a separate brightness value, so that changes in roombrightness can be analyzed independently of color or hue. Alternativecolor spaces (such as YCrCb, CIE Lab, CIE Luv) can separate lightintensity from chromatic changes better than the RGB color space.Processing the chromatic component in those spaces can revealphysiological modulation better than in RGB space, when overall scenelight intensity is changing. Assessing pixel signals based on chromaticchannels in these spaces can increase the robustness of the algorithmand/or increase the range of conditions in which physiological signalextraction is possible. Though the Red/Green/Blue color scheme is oftenpresented here in the examples, it should be understood that other colorschemes or color spaces can be utilized by these systems and methods.

FIGS. 3A and 3B depict five regions of interest—three squares in theforehead, the combination of all three squares together, and onerectangular chest region. In other embodiments, regions of interest canhave various shapes, configurations, or combinations. Examples are shownin FIGS. 5A and 5B. In the embodiment of FIG. 5A, a monitor 500Adisplays an image frame 502, which depicts a region of interest on apatient, in this case a face region, and in particular a forehead region503. The face region is further divided into a grid 504, segmenting theface region into smaller individual regions. Within this grid 504,individual regions of interest 506A, 506B, 506C, 506D, . . . 506N areidentified. The regions of interest 506A-N are regions that includepixels or detected areas that exhibit a physiologic characteristic ofthe patient. A sub-set of the regions of interest can be chosen to focuson a particular physiologic characteristic that is reflected in thepixels in those regions.

In one embodiment, the selected regions of interest (for measurement ofa vital sign) are completely enclosed within the patient region, such asthe face or a smaller area such as the forehead. For example, in FIG.5A, the regions 506A-C are completely contained within the patient'sforehead region 503. No portion of regions 506A-C includes pixelsoutside of the patient's forehead. These regions 506A-C are used toidentify a physiologic signal and calculate a vital sign, such as thepatient's heartrate, as described above. By enclosing the regions withina physiological area, such as the forehead, according to someembodiments, the signal to noise ratio of the desired physiologic signalincreases.

In another embodiment, the selected regions of interest may benon-adjacent to each other, or non-contiguous. For example, in FIG. 5A,non-adjacent regions 506A and 506D may both include pixels that exhibitlarge modulations correlated with the patient's heartrate, as comparedto the other regions. Regions located over large arteries may exhibitlarger modulations with heartrate than other regions, for example. In anembodiment, the intensity signals from regions 506A and 506D areaveraged together to create a combined signal, and the heartratemeasured from that combined signal. Different non-adjacent regions maybe chosen for other vital signs, such as respiration rate or oxygensaturation. In an embodiment, heart rate and oxygen saturation arecalculated from a combined signal from a first group of non-adjacentpixels or regions, and respiration rate is calculated from a differentcombined signal from a second, different group of non-adjacent pixels orregions.

In an embodiment, regions of interest within the image frame areselected based on the modulations exhibited by the pixels in eachregion. Within an image frame, a sub-set of regions may be firstidentified as candidate regions for further processing. For example,within an image frame, an area of exposed skin of a patient isidentified by facial recognition, deduction of a forehead region, userinput, and/or skin tone detection. These areas are identified as theregions of interest for further processing. In an embodiment, facialrecognition is based on Haar-like features (employing a technique thatsums pixel intensities in various regions and differences between sums).A method includes identifying these regions of interest, extractingpixel signals from each region, quantifying the magnitude ofphysiological modulations exhibited by each pixel signal, selectingregions with strong modulations (such as modulations with an amplitudeabove a threshold), combining the selected pixel signals together (suchas by averaging), and measuring a vital sign from the combined signal.In an embodiment, all sub-regions (such as grids) in the image (or aportion of the image, such as a patient region) are processed, and gridcells that exhibit coherent pulsatile components are combined togenerate the pixel signals from which the physiologic measurements aretaken.

Selecting non-adjacent regions enables the system to focus on the pixelsor regions that carry the physiologic signal with the highest signal tonoise ratio, ignoring other areas in the image frame that arecontributing a relatively higher degree of noise, such as pixels that donot vary much with heart rate, but that might vary due to a passingshadow or patient movement. The system can focus on pixels thatrepresent the desired vital sign, thereby increasing the signal-to-noiseratio (SNR) of the analyzed signal. With signals from several regionsavailable, the signals with the strongest SNR can be chosen, and signalswith weak SNR can be discarded. The chosen signals can be combinedtogether to produce a signal with a strong physiologic component.

Referring to FIG. 5A, the size of the cells within the grid 504 canaffect the computation of the resulting pixel signals. If the cells inthe grid are very small (such as 10 pixels by 10 pixels), the number ofcells increases, causing the number of computations and availablesignals to increase. The variability of the signals also increases withvery small cell sizes. For example, a passing shadow or a twitch canaffect a very small area of skin. If a region of interest is whollycontained within that affected area, the signal from that region willbecome noisy. Larger regions provide a degree of spatial smoothing thatreduces susceptibility to such noise, but regions that are too large insize may obscure the physiologic signal. An example of a region of agood size for processing a physiologic signal is approximately onesquare centimeter (though more or less may also be useful—for example awhole forehead may be used, or an individual pixel). If far away fromthe subject, a camera may use less pixels. The selection of region sizealso depends on the resolution of the image, which may depend on theavailable hardware. Moreover, resolution and frame rate may beinter-related, in that increasing resolution may reduce frame rate. Acompromise is necessary between high enough resolution to capture themodulating pixels, and a fast enough frame rate to track thosemodulations over time. Frame rates over 10 Hz are sufficient for cardiacpulses, and over 2-3 Hz for respiration modulations. Frame rates aboveabout 50 or 60 frames per second are generally less subject to aliasingfrequencies introduced by artificial lighting. Sampling from a fewhundred pixels (such as over 200 or over 300 pixels) has been sufficientto isolate a physiologic modulation above ambient noise.

The selected regions of interest can change over time due to changingphysiology, changing noise conditions, or patient movement. In each ofthese situations, criteria can be applied for selecting a pixel, groupof pixels, or region into the combined signal. Criteria are applied toenhance the physiologic signals by reducing or rejecting contributionsfrom stationary or non-stationary non-physiologic signals. Criteria caninclude a minimum SNR, a minimum amplitude of physiologic modulations, aminimum variability of the frequency of modulations (to rejectnon-physiologic, static frequencies), a skew metric (such as modulationsthat exhibit a negative skew), pixels with values above a threshold (inthe applicable Red, Green, or Blue channel), pixels that are notsaturated, or combinations of these criteria. These criteria can becontinually applied to the visible pixels and regions to select thepixels that meet the criteria. Some hysteresis may be applied so thatregions or pixels are not added and removed with too much chatter. Forexample, pixels or regions must meet the criteria for a minimum amountof time before being added to the combined signal, and must fail thecriteria for a minimum amount of time before being dropped. In anotherexample, the criteria for adding a pixel or region to the combinedsignal may be stricter than the criteria for removing the pixel orregion from the combined signal.

For example, in an example involving motion, when the patient turns hisor her head, the regions of interest that previously demonstrated heartrate with the best amplitude are no longer visible to the camera, or maybe covered in shadow or over-exposed in light. New regions of interestbecome visible within the field of view of the camera, and these regionsare evaluated with the criteria to identify the best candidates for thedesired vital sign. For example, referring to FIG. 5A, cells or groupsof pixels at the edges of the forehead region 503 can be added orremoved from the combined signal during motion as they enter and exitthe forehead region. This method enables the monitoring system tocontinue to track the vital sign through movement of the patient, evenas the patient moves or rotates with respect to the camera.

Selected regions may also change over time due to changing physiology.For example, these regions can be updated continually or periodically toremove pixels that do not satisfy the criteria for vital signmeasurement, and add new pixels that do satisfy the criteria. Forexample, as the patient's physiology changes over time, one region ofthe forehead may become better perfused, and the pixels in that regionmay exhibit a stronger cardiac modulation. Those pixels can be added tothe combined light signal to calculate the heart rate. Another regionmay become less perfused, or changing light conditions may favor someregions over others. These changes can be taken into account by addingand removing pixels to the combined signal, to continue tracking thevital sign.

Selected regions may also change over time due to changing noiseconditions. By applying the criteria over time, pixels or regions thatbecome noisy are removed from the combined light intensity signal, sothat the physiologic signal can continue to be monitored via pixels orgroups that are less noisy. These updates can be made continually.

In another embodiment, as shown in FIG. 5B, individual pixels 508A-Nwithin the image frame 502, rather than regions or groups of contiguouspixels, are selected and summed together to produce a signal from whicha patient vital sign can be measured. In FIG. 5B, the patient regionneed not be divided into sub-regions, such as the grid 504 shown in FIG.5A. Rather, individual pixels 508 within the patient region areevaluated, and the pixels that modulate in correlation with the desiredvital sign are selected and summed together. These pixels need not beadjacent or in a near vicinity of each other.

FIG. 5E shows a method for video-based monitoring of a patient's vitalsigns, according to an embodiment. The method includes receiving a videosignal from a video camera at 511. The video signal includes a pluralityof sequential image frames, each image frame having a field of view thatincludes exposed skin of a patient, such as the face or forehead. Themethod includes segmenting a first image frame into a plurality ofregions at 512, and then, for each region, extracting from the videosignal a time-varying color signal at 513. In an example, threetime-varying color signals are extracted from each region, correspondingto red, green, and blue pixel values. The method includes identifying afrequency content of each color signal at 514, and selecting regionsthat have a shared frequency content at 515. The shared frequencycontent is a modulation at a shared frequency. For example, two regionsthat both exhibit color signals that modulate at the patient's heartrate, such as a frequency of 60 beats per minute, are selected. In anembodiment, the shared modulation must pass criteria, such as thosedescribed above, to select the desired regions. For example, anamplitude threshold for the modulation frequency can be applied as acriterion for selecting regions. In an embodiment, the regions thatsatisfy this criterion are non-adjacent to each other; they do not needto be in contact with each other or next to each other on the patient.Rather, regions that exhibit a shared modulation at a physiologicfrequency, above a noise threshold, are selected even if they arelocated at disparate, non-contiguous locations across the patient.

Once the desired regions are selected, the method includes combining thecolor signals of the selected regions at 516, and measuring a vital signfrom the combined color signal at 517, such as measuring heart rate fromthe identified frequency. The vital sign is output for furtherprocessing or display at 518. The calculated vital sign can be added toa long-term running average, or a weighted average, where the weight isbased on quality metrics such as signal to noise ratio or vital signvariability.

The combined light signal can be used to calculate statistics, such asan amplitude of the physiologic frequency (in the time or frequencydomain), a variability of the frequency over time, a variability of theintensity or color of the selected pixels over time, a skew of themodulations, or a signal to noise ratio. Skew is a useful metric becausecardiac pulses tend to have a negative skew. Thus, modulations of pixelsthat exhibit a negative skew may be more likely to be physiologic. In anembodiment, one or more statistics are calculated, and then used toapply a weight to each color signal (from an individual pixel or from aregion) that is being combined. This method results in a weightedaverage that applies more weight to the pixels that exhibit modulationsthat are stronger or more likely to be physiologic. For example, pixelsthat modulate with a strongly negative skew, or a high signal to noiseratio, can be weighted more heavily. The criteria used to select regionscan also be used to assign weights; for example, regions or pixels thatmeet a first, stricter set of criteria may be combined with a first,higher weight, and regions or pixels that meet a second, looser set ofcriteria may be combined with a second, lower weight.

In an embodiment, a weight can also be applied to the vital sign that iscalculated from the combined light signal. Each time the vital sign iscalculated, a weight can be determined based on current quality measuresor statistics from the combined light signal. The newly calculated vitalsign is then added to a longer-term running average, based on theweight. For example, the patient's heart rate can be calculated from thecombined light signal once per second. An associated weight can becalculated based on the criteria applied to the combined light signal.The weight is reduced when statistics indicate that the light signal maybe unreliable (for example, the amplitude of the modulations drops, orthe frequency becomes unstable, or the intensity changes suddenly) andincreased when statistics indicate that the light signal is reliable.

Furthermore, different combinations of pixels (and/or regions) may beselected for different vital signs of the patient. For example, a firstgroup of pixels and/or regions is summed together to produce a signalthat modulates with heart rate, and a second group of pixels and/orregions is summed together to produce a signal that modulates withrespiration rate. This approach is demonstrated in FIGS. 5C and 5D,which each show a light intensity signal over the same span of time fromthe same video signal for the same patient, from different regions, suchas groups of pixels. The pixels chosen for the plot in FIG. 5C exhibitrelatively large fluctuations correlated with the patient's respiration.This is shown by the large baseline modulations 520, with period P1, inthe plotted pixel signal. The frequency of the modulations 520 is thepatient's respiration rate, such as 5-20 breaths per minute. Bycontrast, the pixels chosen for the plot in FIG. 5D do not fluctuate asdramatically with the patient's respiration, but they do fluctuate withthe patient's heart rate, as shown by the modulations 530 with shorterperiod P2. The frequency of these modulations is the patient's heartrate, such as 40-200 beats per minute. These two different plots shownin FIGS. 5C and 5D reflect different vital signs of the patient, basedon the same video stream from the same camera taken over a single periodof time. By creating combined pixel signals from appropriately selectedpixels or regions, various physiologic signals emerge from the videoimages.

Accordingly, in an embodiment, a method is provided for measuringdifferent vital signs from different regions. These groups can includeindividual pixels, disparate pixels, contiguous regions, non-contiguousregions, and combinations of these. Pixels combined into one groupexhibit a common modulation, such as a frequency of modulation of coloror intensity. For example, heart rate can be measured from the frequencyof modulation of a first group of pixels, and respiration rate can bemeasured from the frequency of modulation of a second group of pixels.Oxygen saturation can be measured from either group; in one embodiment,oxygen saturation is measured from the pixels that show strongmodulation with heart rate. Specifically, oxygen saturation is measuredas a ratio of ratios of the cardiac pulsatile components of two of thesignals (such as Red and Green, or Red and Blue) (as described in moredetail below).

In an embodiment, a user can view a video image, specify a region ofinterest, and drag and drop the region across the video image to viewchanges in modulations in real-time. For example, referring to FIG. 5B,a monitor 500B displays a video image 502 that accepts inputs from auser. A user can use mouse pointer 509 (or other input) to highlight afirst area 507A, and view the resulting pixel signals such as the signalshown in FIG. 5C and vital signs measured from that signal. The user canthen drag and drop the area of interest to a second area 507B and viewthe resulting signal and vital signs, such as the signal shown in FIG.5D. In this way, the user can view in realtime how the modulations ofthe signal change based on the selected area of interest. In area 507A,the video signal shows strong respiration modulations (see FIG. 5C),while in area 507B, the video signal shows strong cardiac modulations(see FIG. 5D). The user can view the video signal in real-time as itmoves along the path from 507A to 507B, to see how the modulationschange as the region of interest moves. The user can also view the pixelsignals shown in FIGS. 5C and 5D at the same time, to evaluate differentvital signs from different regions of interest, at the same time.

A method for monitoring a patient by viewing these different modulationsacross different regions of interest is outlined in FIG. 5F. The methodincludes displaying a video signal at 521, and receiving a first userinput identifying a region of interest within the video image at 522.The method includes extracting a color signal from the region ofinterest at 523, and displaying the color signal at 524. The method thenincludes receiving a second user input that moves the region of interestalong a path (such as from 507A to 507B in FIG. 5B) at 525. The methodincludes continually updating the location of the region of interest inaccordance with the second user input at 526, continually updating thecolor signal from the region of interest at 527, and displaying theupdated color signal at 528. This enables a user to dynamically changethe region of interest and view the resulting extracted video signal, todynamically see the modulations at any point in the field of view. Inaddition to displaying the color signal, vital signs can be calculatedfrom the moving region of interest and displayed to the user.

In an embodiment, a video-based method for measuring a vital sign of apatient includes receiving a video signal, displaying on a displayscreen an image frame from the video signal, and receiving from a user afirst user input that identifies a location of a first region ofinterest within the image frame. The method includes extracting from thefirst region of interest a first color signal comprising a time-varyingintensity of one or more pixels in the first region, detecting amodulation of the first color signal, and measuring a first vital signof the patient from the modulation of the first color signal. The firstvital sign and/or the modulation may be displayed. The method alsoincludes receiving a second user input indicating that the location hasbeen moved to a second, different region of interest. The method thenincludes extracting from the second region of interest a second colorsignal, detecting a modulation of the second color signal, measuring asecond vital sign of the patient from the modulation of the second colorsignal, and displaying the modulation and/or second vital sign. In anembodiment, the method includes identifying a plurality of intermediateregions of interest along the path from the first to the second regionof interest, extracting an intermediate color signal from one of theintermediate regions, and displaying on the display screen a modulationof the intermediate color signal.

In an embodiment, the desired pixels are chosen based on a ratio ofmodulations in the pixel signals. A ratio R is defined as A_(H)/A_(R),where A_(H) is the cardiac pulse amplitude, and A_(R) is the respiratorymodulation amplitude. The region where R is maximum (or above a suitablethreshold) can be used to determine heart rate, and the region where Ris minimum (or below a suitable threshold) can be used to determinerespiratory rate. A method according to this embodiment is shown in FIG.6A. Regions may be increased or decreased in size, or discrete regionsor pixels combined together, to obtain a combined pixel signal with anoptimal or desired ratio R.

As discussed above, a region of interest can be formed based on pixelsthat modulate with the patient's heart rate. Heart rate can then becalculated from the frequency content of that pixel signal. An examplemethod for calculating heart rate is shown in FIG. 10B. The methodincludes capturing video, acquiring and averaging color signals (shownas pR, pG, and pB for “photoplethysmogram” red, green, and blue) withina well-perfused ROI, de-noising the signal, performing an FFT (fastFourier transform) operation over a sliding time window (such as 20seconds) to identify frequency components of the signals, finding peakfrequencies, and accumulating peaks over a period of time (such as onesecond). De-noising includes filtering the signal to remove noisesources and frequencies outside of a known physiologic range. Examplesof filtering operations to remove noise are described below withreference to FIGS. 17A and 17B. In accumulating peaks, the method mayadd frequencies multiple times based on their relative height, and mayadd harmonics of already-added frequencies only once. Frequency peaksare added to the accumulator at the frame rate, such as 25-30 times persecond.

Then, once per second, the method finds a median frequency from theaccumulated ones, and determines heart rate from the median frequency.The determined heart rate is added to an ongoing average, and thenposted for display. As noted in the figure, different averagingtechniques may be employed for the externally-posted heart rate as wellas for an internally-maintained running average, such as to applyadditional smoothing to the externally-posted heart rate. When multiplepeaks are present, additional filtering can be applied to determine themost likely heart rate. For example, frequency peaks outside of knownphysiologic limits for heart rate (such as below 40 or above 250 beatsper minute) are rejected. Knowledge of the patient's previous heart rateis also useful, as the heart rate is unlikely to jump a large amount(such as 2.5% of the current heart rate, or another percentage, or avalue such as 15 or 20 beats per minute) within 1 second, so suchfrequency peaks can be rejected as noise. Within the acceptable peaks,the strongest peak is selected as the patient's heart rate. When themedian frequency is rejected as noise, the previously-calculated heartrate is held for one additional second, while the next group of peaks isaccumulated, and the range for an acceptable heart rate is increased.When the new group of peaks is assessed, a median frequency picked, anda new heart rate calculated, the acceptable range is re-set to itsnormal size, around the new average heart rate. The same method can beused to determine respiration rate, within different frequency rangesand time windows, applied to the same or different pixel signals.

FIG. 10B also includes a cross-correlation process that cross-correlatesthe frequency spectrums of the three color signals to amplify theresults. All four resulting spectrums are analyzed to select andaccumulate peaks. A cross correlated spectrum can be calculated bymultiplying or summing existing spectrum together. An individualspectrum can be scaled before being combined based on signal quality.For example, because most RGB cameras have twice the number of greenpixels compare to red and blue ones, the Green signal is usually betterand can be weighted above Red and Blue. This method can follow thestrongest peaks around the spectrum over time, as the patient'sphysiology (such as respiration rate and heart rate) changes.

In an embodiment, a method for monitoring a patient's heart rateincludes generating a video signal from a video camera having a field ofview encompassing exposed skin of a patient. The video signal includes atime-varying intensity signal for each of a plurality of pixels in thefield of view. The method includes combining the intensity signalswithin a region of the field of view to produce a regional intensitysignal, and transforming the regional intensity signal into thefrequency domain to produce a regional frequency signal. The region maybe selected based on a strength of modulations of intensity signals inthe region. The region may include non-adjacent areas or pixels. Over asliding time window, peaks in the regional frequency signal areidentified, and then over a period of time (such as one second), theidentified peaks are accumulated. The method includes selecting a medianfrequency from the identified peaks, and updating a running averageheart rate of a patient, which includes converting the median frequencyinto a measured heart rate and adding the measured heart rate to therunning average. The updated average heart rate is output for display.The method may also include removing identified peaks from theaccumulated peaks when they reach an age limit. The method may alsoinclude discarding frequency peaks outside of a physiologic limit, ordiscarding the measured heart rate when it differs from the averageheart rate by more than a defined amount. The method may also includediscarding frequency peaks if they are sub-harmonics of alreadyidentified peaks.

An example frequency transform of a pixel signal from a region ofinterest is shown in FIG. 10A. This figure shows three (Red, Green, andBlue) pixel signals over time and the FFT operation, which is applied toa 20-second window of the cross-correlated spectrum of all threesignals. The FFT shows a strong peak at 66.0 beats per minute. In themethod of FIG. 10B, these peaks are added to the frequency accumulator,the median peak is identified, and the patient's heart rate calculatedfrom the median peak.

The non-contact video monitoring system provides many benefits overtraditional contact sensors, and also enables monitoring in new anddifficult situations. In one example, the non-contact video-basedmonitoring system can be used to measure vital signs in patients who arenot able to tolerate a contact-based sensor, such as patients with skintrauma. These patients could include burn victims, or patients withother sensitive skin conditions. In another example, the non-contactvideo-based monitoring system can be used to measure multiple patientsat the same time (see FIG. 2B). A method for monitoring two or morepatients at the same time includes orienting the field of view of thecamera to encompass two or more patients. In an embodiment, the camerais oriented such that the field of view encompasses exposed skin of eachpatient, and groups of pixels that exhibit physiologic modulations areidentified for each respective patient. A single camera system can thenbe used to measure vital signs from multiple patients, such as patientson a general care floor, or to track movement of patients within a roomor ward.

The vital signs measured from the video signal can be used to triggeralarms based on physiologic limits (for example, high or low heart rate,SpO2, or respiration rate alarms). The video signals, the measured vitalsigns, and triggered alarms can be used by clinicians to identifypatients in distress, provide clinical intervention, apply a treatment,support a diagnosis, or recommend further monitoring. The vital signsmeasured from the video signals may be further processed to arrive at afinal value that can be displayed or compared to alarm limits. Furtherprocessing may include adding the vital sign to a running average (suchas an infinite impulse response filter) to smooth out variability,rejecting outlier vital sign measurements that are not supported byknown physiological limits (such as a newly calculated heart rate thatvaries by more than a physiologically expected amount, as discussedabove), increasing or decreasing a weight applied to the vital sign,calculating statistics relating to the vital sign, or other processingsteps. The result is a final number, derived from the vital signmeasurement from the intensity signal, and this final derived number canbe displayed, stored, or compared to alarm limits.

Oxygen Saturation

According to an embodiment of the invention, the Red/Green/Blue pixelstreams from identified areas of the patient's exposed skin can be usedto determine arterial oxygen saturation (SpO2). Traditional pulseoximeters employ contact-based sensors, which include two emitters(typically light emitting diodes, LED's) and a photodetector. Theemitters are positioned on the sensor to emit light directly into thepatient's skin. The emitters are driven sequentially, so that light ofeach wavelength can be separately detected at the photodetector,resulting in two time-varying light intensity signals. The wavelengthsare chosen based on their relative absorption by oxygenated hemoglobinin the blood. Typically one wavelength falls in the red spectrum and theother in infrared. The patient's arterial oxygen saturation can bemeasured by taking a ratio of ratios (ROR) of the two signals—that is,by taking a ratio of the alternating component (AC) of each signal toits direct, non-alternating component (DC) and dividing the red ratio bythe infrared ratio.

In a video-based system, the Red/Green/Blue pixels or regions detectedby the camera provide three light intensity signals that potentially canbe used in a similar ratio of ratios calculation, such as by dividingthe ratios of any two of the three signals. However, many standard videocameras do not detect light in the infrared wavelengths. Moreover, formany video cameras, the wavelengths of light detected in each of theRed, Green, and Blue components overlap. For example, the video camera214 (see FIG. 2A) may include an image sensor with broad spectrum red,green, and blue detectors. The wavelengths detected by these detectorsoverlap, and are not chosen specifically for their relative absorptionby oxygenated hemoglobin. As a result, measuring a ratio of ratios fromtwo of the three signals does not provide an absolute, calibrated SpO2value. However, such a ratio of ratios can be used to track the trend ofthe patient's actual SpO2 value.

Such a trend is shown in FIG. 6B. The top plot in FIG. 6B shows an SpO2value from a calibrated, contact-based pulse oximeter. It also shows twoheart rate signals, one taken from the same pulse oximeter and the otherfrom a video signal. It is readily apparent that the video-based heartrate signal tracks the oximeter-based heart rate signal very closely,providing good absolute correlation.

The bottom plot in FIG. 6B shows three different SpO2 values from avideo signal, one for each pair of signals. The top trace is from aratio of ratios calculation of the Red and Green signals, the middle isthe Red and Blue signals, and the bottom is the Green and Blue signals.These three traces can be compared with the calibrated SpO2 valueplotted above, from the conventional contact pulse oximeter. It is clearfrom FIG. 6B that all three traces correlate with the calibrated SpO2plot, in that they trend up or down in proportion to the calibrated SpO2plot. However the absolute values (shown in the y-axes in FIG. 6B) ofthe video-based SpO2 traces do not match the calibrated SpO2 valueitself. The calibration of the SvidO2 against SpO2 may be performed bylinear regression, whereby the coefficients of the regression model areapplied to the SvidO2 to estimate the absolute SpO2 values.

In an embodiment, the video-based SpO2 measurement is used as a trendindicator, rather than as a measurement of an accurate SpO2 numericalvalue. For example, it is apparent from the Blue-Red trace that the SpO2value remains stable until time t1, begins to change at time t1,decreases until time t2, remains stable at low oxygenation until timet3, increases again until time t4, and thereafter remains stable again.The Blue-Red trace can thus be used as a trend indicator, to provide analert that the patient's SpO2 value is changing, and can even indicatewhether the SpO2 value is increasing or decreasing, and an indication ofthe rate of increase or decrease. This information can be used toprovide an early warning to a caregiver that the patient needsattention, such as by attaching a traditional contact-based pulseoximeter to obtain a numerically accurate reading of the patient's SpO2value which can be used to determine a diagnosis or treatment.

In another embodiment, the SpO2 value measured from a pair of theRed/Green/Blue pixel streams is calibrated to an accurate numericalvalue. Calibration can be done by comparing the video-based SpO2 valueto the value from a reference contact-based oximeter, to identify anoffset between them. This offset is used to determine a scaling factorthat is applied to the ROR calculation from the video signal. Forexample, the scaling factor can be a coefficient multiplied to the videoROR, or an offset added or subtracted from the video SpO2, or both. Thisoffset and/or coefficient can be used until the next recalibration.Recalibration can be done when a set time has expired, or when the videoSpO2 trend shows a marked change in SpO2.

FIG. 7 shows a method of calibrating a video-based SpO2 measurement,according to an embodiment of the invention. The method includesperforming a spot check with a contact oximeter at 701, comparing theoximeter SpO2 to the video SpO2 (also called S_(vid)O2) at 702, anddetermining the calibration between the two values (such as an offset,scaling factor, and/or coefficient) at 703. The method then includesmeasuring SpO2 from the video signal with the calibration at 704. At705, a timer is used to prompt re-calibration. For example, the timermay be set to expire in 15 minutes, or one hour, or two hours, or othertime durations desired by the caregiver. If the time has expired, themethod returns to 701; if not, the method continues to 706, where thevideo SpO2 value is compared to a threshold to identify changes. If thevideo SpO2 value crosses the threshold, the method includes sounding analarm (such as an audible sound and/or a visible alert) at 707, andprompting re-calibration at 701. If not, the method returns to continuemeasuring at 704. The threshold used to detect a change at 706 can beset by the caregiver to identify changes in video SpO2 that may indicatea clinically significant change in the patient's physiology, for furtherdiagnosis or treatment.

When calibration or re-calibration is not available, the monitor maycontinue to calculate video SpO2 to identify trends. The trend from thevideo SpO2 may be used to trigger an alarm when the trend shows thatSpO2 is rapidly changing or has crossed an alarm threshold. Clinicallyrelevant patterns (such as repeated desaturations) may also be detectedfrom the video SpO2 signal, between or in the absence ofre-calibrations.

When the video-based SpO2 value is calibrated to an accurate measure ofoxygen saturation, it can be tracked from there to measure the patient'sactual SpO2 value. An example of this is shown in FIG. 8, which plotstwo SpO2 values, one from a traditional contact-based pulse oximeter,and the other from a calibrated video-based pulse oximeter. Thevideo-based SpO2 value in this example is taken from the Red and Greensignals, and then calibrated with an absolute SpO2 value as describedabove. Once calibrated, it is clear from FIG. 8 that the video-basedSpO2 value tracks the patient's absolute SpO2 value closely. The datapresented in FIG. 8 was collected during a clinically-relevantdesaturation event in which the subject's oxygen saturation dipped andthen recovered.

Though the video-based SpO2 measurement can be calibrated from acontact-based pulse oximeter, the video-based SpO2 measurement mayexhibit different behavior over time, as compared to a traditionalcontact-based oximeter. These differences may arise due to thedifferences in filtering characteristics between the contact-basedoximeter and video camera, and/or differences in the light waveformsdetected by a remote video as compared to a contact-based sensor, and/orother factors. As an example, the light detected by a remote videocamera may be reflected from a shallower depth within the patient'stissue, as compared to contact-based oximetry, which utilizes a contactsensor to emit light directly into the patient's tissue. This differencein the light signal can cause the morphology of the video-detectedwaveform to differ from a contact-based waveform. As another example,the light detected by a remote video camera is more susceptible toambient light noise incident on the surface of the region beingmonitored.

As a result, the SpO2 measurement from the video-detected waveformexhibits some differences from the contact-based SpO2 measurement, evenwhen the two are first calibrated together. An example of this behavioris evident in FIG. 8. Between times t1 and t2, the subject's oxygensaturation drops and then recovers to a baseline level BL. Bothwaveforms track this trend, but the video-based measurement is slowerthan the contact-based measurement to return to baseline. The result isa difference, labeled ΔS (delta saturation) between the twomeasurements. Because this behavior of the video-based measurement isknown, it can be corrected for, by adjusting the value upward during anincreasing trend. This adjustment can be tailored based on empiricaldata. An adjustment may be made by finding the relationship (mapping)between the video-based SpO2 and the contact-based (oximeter) SpO2. Thisrelationship may then be coded within the video system to mimic theoximeter-based SpO2.

In an embodiment, the video-based non-contact monitoring systemidentifies acute hypoxia in monitored patients, by identifying episodesof decreased oxygen saturation. The system provides continuousmonitoring of vital signs such as video-based SpO2, rather thandiscrete, periodic spot-check readings. This continuous monitoring, viaeither trending or calibrated video SpO2, enables the system to identifyclinical conditions such as acute hypoxia, and repeated interruptions inairflow.

In an embodiment, a monitoring system is programmed to take certainsteps including activating alarms or messages when a suitablephysiologic signal is not ascertainable in the field of view. Forexample, in an embodiment, a processor acquires a physiologic signal (asdescribed above), and determines a physiologic parameter from thesignal. However the signal may be lost when the patient moves out of thefield of view, or moves in such a way that a physiologic region (such asexposed skin) is not visible, or moves too quickly for accuratetracking. The signal may also be lost if another person or item movesinto the field of view and blocks the camera's view of the patient, orif the room becomes too dark (such as if room lights are turned off atnight). In any of these or similar situations, the processor starts atimer counting down, and holds the previous value of the calculatedphysiologic parameter. After a short duration, the processor may send analert message to be displayed on a screen or otherwise notified to aclinician, to indicate that the signal has been lost and the parametervalue is held frozen. If the timer expires, the processor can then soundan alarm or other notification, such as an escalated message orindicator, and remove the frozen physiologic parameter value (orotherwise indicate that it is a previous value, no longer beingupdated). This can be a system-level alarm or notification, whichindicates a problem with the signal acquisition, as distinguished from aphysiologic alarm (that would indicate a physiologic parameter of thepatient crossing an alarm threshold). This alarm or notification can bea message stating that the room lights have been turned off, or thepatient has exited the field of view, or the patient is obscured in thefield of view, or the patient is moving, or other applicablecircumstance.

This message can be displayed at a remote station (such as a nursingstation at a hospital) or on a remote, wireless device (such as asmartphone, tablet, or computer). Additionally, at a central monitoringstation (such as a nursing station at a hospital), where display screensdisplay information about multiple different patients, the video-basedmonitoring system can alert the central station to highlight anindividual patient. For example, the remote monitoring system can sendan alert or flag based on a change in condition (a system-level alarm, aphysiologic alarm, an activity level of the patient, etc.), and thecentral station can then enlarge the video stream from that particularcamera. This enables the caregivers at the station to quickly assess thesituation in the room and determine if urgent action is needed.

In an embodiment, the processor identifies or is informed that aclinician or caregiver is interacting with the patient, and theprocessor temporarily halts dynamic tracking of the intensity signaland/or temporarily halts calculation of a physiologic parameter from theintensity signal. This step is taken because such interaction interfereswith the camera's view, rendering the light intensity signals more noisyand less reliable. When the interaction is finished, the processorresumes its remote monitoring of the patient.

Ambient Light

As mentioned previously, changes in ambient light in the camera's fieldof view can obscure the subtle variations in the detected pixel streamsthat are attributable to the patient's physiology. In an embodiment ofthe invention, a video-based monitoring system includes a calibrationstrip that can be used to identify and correct for these changes inambient light. A calibration strip 1100 according to an embodiment isshown in FIG. 11. The calibration strip 1100 is sized to fit on thepatient (such as along the patient's forehead) and within the field ofview 1102 of the camera. In an embodiment, the calibration strip 1100includes a scale which displays a range of values for measurement, suchas a greyscale with two or more grey or white hues; or a color map withtwo or more different colors. The scale can include a continuousspectrum of varying intensity and/or color, or it can include a set ofdiscrete areas each with a different color and/or intensity. In oneembodiment, the color map includes one or more known skin tone colors,which are then compared to exposed skin of the patient to identify anapproximation of the patient's skin tone, which can then be used toadjust the exposure settings if the camera based on the light intensityof the skin. These values may vary along one (e.g. longitudinal) or twodimensions of the calibration strip. For example, the calibration strip1100 shown in FIG. 11 includes a grid 1104 with four different discreteregions 1106A, 1106B, 1106C, and 1106D. Each region displays a differentintensity and/or color. The colors have a known chromatic value, whichallow for the colors in the captured video image to be color balanced tomake corrections. Another example is a strip with a grey square or othershape. The intensity of the patch or portions of the patch (such as agrey square) identified in the video image can be used to adjust theexposure settings on the camera. In an embodiment, the calibration striphas a matte finish to reduce reflected light.

In an embodiment, a calibration strip includes spaces that are Red,Green, Blue, and white. This strip provides a reference for colorbalancing the region of interest on the patient. For example, if thewhite space from the calibration strip appears with a green hue on theimage, then the region of interest can be color balanced to remove thegreen skew. This can be particularly helpful for SpO2 measurements.

FIG. 9 shows an embodiment of a video-based method of measuring apatient's vital sign. The method includes providing a calibration stripcomprising a substrate with a visible scale for viewing by the videocamera in the same field of view as the patient, at 901. The methodincludes detecting, by the video camera, a first light signal reflectedfrom the scale and a second light signal reflected from exposed skin ofthe patient at 902, and adjusting a calibration of the video camerabased on a measurement of the first light signal at 903. The methodincludes applying the calibration to the second light signal at 904,measuring a vital sign of the patient from the calibrated second lightsignal at 905, and outputting the measured vital sign at 906 for furtherprocessing or display. The scale may be a greyscale or a color map. Themeasurement of the first light signal can be a measurement of anintensity of the light reflected from the scale, such as a portion ofthe color map or the greyscale.

The method includes monitoring the measured intensity to detect changesin ambient lighting in the room at 907. For example, at a later time,the system measures a second intensity that differs from the firstintensity by a defined amount (such as an amount exceeding a threshold,to avoid excessive adjustments), such as a sudden increase or decreasein intensity due to room lights being turned on or off. If this changemeets the defined amount or threshold (or other criteria), the methodpasses back to 903 to adjust the calibration based on the secondintensity, such as by adjusting a coefficient in proportion to thedefined amount or the new measured intensity. The coefficient is appliedto the light intensity signal, to re-normalize the signal to the secondintensity. The coefficient is applied by the camera in generating thered, green, and blue pixel signals. Otherwise, the method returns to 905to continue monitoring. This method enables the red, green, and bluesignals to be normalized to reference values, to better identify thephysiologic signals even in differing light conditions. Thisnormalization also enables two different cameras, monitoring twodifferent patients, to be adjusted to the same reference color orbrightness values, so that the vital signs or other measurements fromthe light signals can be compared to each other, without skew due todifferent camera hardware or light conditions.

In an embodiment, the scale on the patient includes a color map with aplurality of colors, and the measurement of the first light signal is ameasurement of a color value of one of the plurality of colors. Then,the calibration is adjusted by comparing the color value to a referencecolor value and identifying a difference. Baseline color values from thescale can also be stored at a first time, and then the calibration canbe adjusted later based on comparisons of new measurements (of lightfrom the scale) to the stored baseline color values, such as when thenew measurement deviates from the baseline by a defined amount. When asecond video camera is used in the same room, the second camera can becalibrated based on the same reference, baseline, or other values usedfor the first camera.

In an embodiment, a calibration strip includes a white space, and thesystem measures the brightness of that white space in the image. Thisbrightness indicates the amount of light hitting that region. The whitespace is monitored for changes in brightness, which may indicate apassing shadow or change in lighting conditions, including changes dueto movements outside the field of view that change the amount of lightreflected onto the region of interest. The color signals from the regionof interest can then be filtered according to the changes in brightness,to continue tracking SpO2 (or another vital sign) during transientchanges in lighting, such as due to motion in the room. This can bedone, for example, with an adaptive filter based on the reference signalfrom measurement of the white space. Average light intensity within theidentified white space can be used as a baseline to compensatenon-physiological changes in the sampling regions. Alternatively, thecolor signals and/or vital sign measurements can simply be discardedduring these transient changes.

In an embodiment, a calibration strip includes a graphic with highcontrast, such as a dark dot, cross or circle on a white space, or agrid of colors. The system can track this high contrast shape to trackmovement of the patient. For example, the system can track the positionand orientation of the high contrast graphic, and can generate a motionsignal that tracks that movement. The motion signal may be a transformthat maps the movement of the graphic. The same transform is applied tothe region of interest in the patient, to track the movement of thatregion. If the transform reveals that the region of interest has exitedthe field of view, then a new region of interest is identified, based onthe desired vital sign. Further, limits can be placed on the allowablerate of motion (such as angular rotation limits), and if the limits areexceeded, the color signals and/or measurements from the region ofinterest can be discarded.

In another embodiment, the calibration strip includes light emittersthat emit light of selected wavelengths into the patient, but withoutthe detector of traditional contact-based oximetry sensors, and withoutany transmitter for transmitting detected light. For example, acalibration strip 130 according to another embodiment is shown in FIG.12. In this embodiment, the calibration strip 130 includes an adhesivepatch 132 that is sized to fit within the field of view of a non-contactcamera, such as on the patient's skin. The patch 132 includes a topsurface 134 that faces away from the patient, opposite a bottom surface136 that faces toward and contacts the patient. The top surface 134carries a scale or graphic 138. The bottom surface carries an adhesive140 that removably adheres to a patient's skin. The patch 132 alsoincludes two emitters 142, 144 coupled to a battery 146 and amicroprocessor 148. When the patch is placed on a patient's skin, theprocessor 148 drives the emitters 142, 144 to emit light sequentiallyinto the patient's skin. The processor drives the emitters in afour-part sequence, in which the first emitter is on, then both emittersare dark, then the second emitter is on, and then both emitters aredark. This sequence is repeated at high frequency, such as 15-30 Hz.

However, notably, the patch 132 does not include a photodetector or anytype of transmitter. Rather, the detector is a non-contact video cameraviewing the patient, as described above. The video camera records imageframes that include at least the portion of the patient's skinsurrounding or near the patch 132. Light from the emitters travelsthrough the patient's tissue and out through this portion of thepatient's skin, such that it can be detected by the video camera. Thissystem is a hybrid approach, employing contact-based emitters and anon-contact, remote detector. The system benefits from having dedicatedlight emitters at chosen wavelengths (for example, a narrow range of redand green wavelengths), creating a stronger physiologic signal in thedetected image frames, while at the same time avoiding the drawbacks ofa tethered sensor system. The patch 132 does not have any cables orwires connecting it to a monitor, nor any wireless communication. Thepatch 132 does not require any communication at all between itself andthe camera or the monitor (such as the camera 214 and monitor 224 inFIG. 2A). As a result, the patch can omit components such as a wirelesstransmitter or receiver and supporting components such as batteries forthose devices. The processor 148 carried by the patch can operate atvery low power, operating only to drive the emitters 142, 144 and not toprocess or transmit any detected signal. The processor and emitters canbe powered by a small battery 146. The patch is also small andlightweight, making it relatively comfortable for the patient to wear,and it does not interfere with the patient's mobility. The camera maybegin monitoring the patient's vital signs automatically when it detectsthe emitted light, or it may be turned on by a caregiver.

It should be noted that the scale 138 shown in FIG. 12 is optional. Inan embodiment, the patch 132 omits the scale 138 on the top surface, andis not used as a calibration strip. In another embodiment, the patch 132includes a single color on the top surface, such as white, for use inmeasuring brightness and detecting passing shadows.

Independent Component Analysis

Due to the exposure of the camera detector to significant ambient lightnoise, the video-based system employs new approaches to filter theambient light noise and identify the physiologic signal from which thepatient's vital sign can be measured. An approach for filteringaccording to an embodiment of the invention is demonstrated in FIGS.13-16. In this embodiment, independent component analysis (ICA) is usedto decompose the Red, Green, and Blue pixel streams into individualcomponents. ICA is a filtering method that, based on certainassumptions, separates input signals (also called source signals) intoseparate, independent components that are mixed together in the inputsignals. The ICA method is described in detail in the following paper:Hyvärinen, A., & Oja, E. (2000). Independent component analysis:algorithms and applications. Neural networks, 13(4), 411-430.

In the context of video-based monitoring, the source signals are theRed, Green, and Blue pixel streams, and the independent components arethe heart rate and the noise. Referring to FIG. 13, the source signalsare shown in the three plots on the left, with the Red pixel stream ontop, Green in the middle, and Blue on the bottom. These source signalsare decomposed via an ICA method into three separate, independentcomponents, shown on the three plots on the right (labeled Component 1,Component 2, and Component 3).

As shown in FIG. 13, Component 1 exhibits a repeating pattern ofmodulations at a relatively steady frequency. Component 1 is constructedfrom the portions of the source signals that modulate at that frequency.In this case, the frequency of the modulations in Component 1 representsthe heart rate of the patient. The contributions of the patient's heartrate to each source signal have been pulled together and combined intothe waveform of Component 1, creating a waveform that identifies theheart rate more clearly than any single source signal did. The patient'sheart rate can be measured from the primary frequency of Component 1.

Still referring to FIG. 13, Components 2 and 3 are relatively moreerratic, and do not exhibit a clear primary frequency. These componentscapture the noise that corrupted the Red, Green, and Blue sourcesignals. Each of the source signals represents a different mixture orcombination of Components 1, 2, and 3.

By utilizing ICA to decompose the source signals, an underlyingphysiologic signal such as heart rate or respiration rate can beidentified. As discussed above, different groups of pixels or regionscan be selected to measure different vital signs, such as heart rate andrespiration rate. FIG. 13 represents the source signals from a firstgroup of pixels or regions that modulate with the patient's heart rate.These signals are decomposed via ICA to arrive at a relatively cleanheart rate signal in Component 1. A different group of pixels or regionsthat modulate with respiration rate can also be decomposed via ICA toarrive at a relatively clean respiration rate signal. Another region maybe decomposed via ICA to arrive a pulsatile signal that demonstrateperfusion status of the patient (such as by Delta POP or DPOP, bymeasuring the variations in amplitude of the pulses at the top andbottom of the baseline modulations). These vital signs may be measuredfrom the same region or different regions.

In FIG. 13, Component 1 exhibits the most regular frequency, as shown bythe plotted vertical lines. Vertical lines are placed in the plots ofComponents 1, 2, and 3 at each local maximum in the waveforms. Thecomponent with the most regularly spaced vertical lines is chosen as thecomponent that represents the patient's heart rate. In FIG. 13, this isclearly Component 1.

FIG. 14 shows the results of an ICA method applied to a video stream tomeasure a patient's heart rate. The figure shows heart rate calculatedby a traditional contact-based oximeter (solid line) as well as heartrate from an ICA filtered video stream (x's) from the same subject overthe same time duration. The ICA-based heart rate shows good correlationwith the traditional oximeter values.

After decomposing the source signals via ICA to identify a physiologiccomponent (such as heart rate), that component can then be used tofilter the original input signals, as shown in FIG. 15. FIG. 15 showsthree traces—Component 1 on top, the Red source signal in the middle,and the Green source signal on the bottom. The vertical lines marked onthe local maximums of Component 1 are projected onto the Red and Greenpixel streams. The locations of these projections signify heart beats inthe Red and Green pixel streams, even when these source signals arecorrupted by noise. The ICA-derived heart rate signal of Component 1 canbe used to identify the location of individual pulses in the sourcesignals. The ICA technique finds the best representative pulse signal,which can then be used to locate pulses in the original Red and Greensource signals.

FIG. 16 depicts a flowchart of a method for measuring a vital sign of apatient with ICA, according to an embodiment. The method includesreceiving a video signal from a video camera at 1601, and extractingfrom the video signal the source signals at 1602, such as time-varyingred, green, and blue signals. The method includes performing ICA togenerate a component signal having a primary frequency at the heart rateof the patient at 1603. Performing ICA involves decomposing at least twoof the source signals into component signals, and selecting thecomponent signal that exhibits the contribution of the patient's heartrate, as explained above. The method then includes identifying, in theselected component signal, an individual pulse representative of anindividual heart beat at 1604. The method includes locating acorresponding portion of at least two of the red, green, and blue sourcesignals at 1605. This can be done by determining a fiducial in thecomponent signal (e.g. the maxima, minima, peak of the first derivative,etc.), and using this fiducial to identify a corresponding pulse orlocation in the source signals. Then, for each of those two sourcesignals, the method includes adding the located portion to a weightedaverage pulse at 1606. This produces at least two of a red weightedaverage pulse, a blue weighted average pulse, and a green weightedaverage pulse. The method then includes measuring blood oxygensaturation of the patient from the weighted average pulses of the twosignals at 1607 (such as by measuring an ROR, and computing SpO2 fromthe ROR). Heart rate can also be measured from the primary frequency ofthe component signal. The vital signs are output for further processingor display. In an embodiment, the method also includes triggering anaudio beep at 1608 in synchrony with the individual pulse identified inthe component signal, or in synchrony with the located correspondingportion of one or two of the color signals. This audio beep signifiesthe occurrence of a cardiac pulse. Instead of an audio beep, otheraudible or visual alerts may be triggered or displayed.

The ICA-derived pulsatile component signal is thus used as a trigger toinform the processor where to look in the original signals for relevantphysiologic information. In turn, this trigger can be used to control anensemble averaging method, in which sequential pulses are averaged withpast pulses to create a smoother average cardiac pulse for each sourcesignal. The ICA-derived trigger may also be passed to another medicaldevice, such as a pulse oximeter, blood pressure monitor, or othermonitor or processor, to inform that device that a cardiac pulse hasbeen detected and the time or location of that pulse.

Noise Reduction

Another way to address noise is to identify non-physiologic peaks withinthe frequency domain, and remove those from the video signals. Twomethods for identifying non-physiologic peaks are summarized here.

In one method, in the frequency domain, peaks are identified that remainstationary over a duration of time. Over a sufficient period of time(long enough for a few cycles of the vital sign—for example, 5-10seconds for heart rate, or 20-30 seconds for respiration rate), peaksthat remain stationary are likely to be non-physiological, such as peakscaused by aliasing from flickering room lights, while physiologic peaksshould move and shift with the patient's state. A frequency transformsuch as an FFT can be performed over different time durations (such asdifferent window sizes), and the frequencies that remain stationary, byappearing the same regardless of window size, are likely to benon-physiological. These identified frequencies can be removed byfiltering. A flowchart illustrating this method is shown in FIG. 17A. Inan embodiment, the method includes performing a frequency transform overfirst and second time windows of different sizes (different timedurations) at 1701. The method includes comparing frequency peaks in thetransforms at 1702, and identifying stationary frequency peaks at 1703.The method then includes filtering the video (source) signal(s) toremove the stationary frequency at 1704.

The number of window sizes, and their sizes relative to each other, canbe varied to achieve a desired result. In an embodiment, two differentwindow sizes are used, one 20 seconds in duration and the other 10seconds in duration. In another embodiment, three window sizes are used,such as 20, 10, and 7 seconds each. This analysis can be done on eachpixel signal individually, to remove identified frequencies from eachsignal, or it can be done on one signal and then the identifiedfrequencies can be removed from all signals.

In another method, in the frequency domain, peaks are identified thatmove based on frame rate. Frequency peaks that move to another positionor disappear when the video frame rate is adjusted may be taken asnon-physiological, because physiologic modulations do not disappear ormove instantaneously based on the video characteristics. In anembodiment, the frame rate sweeps at a constant sweep rate over a rangeof frequencies, or moves along a trajectory (such as a first frame ratefor a first time duration, then a second frame rate for a second timeduration, etc), and frequency peaks that move with that sweep ortrajectory are considered non-physiological. Frequency peaks that moveat the sweep rate are particularly suspect and can be removed. The speedof the sweep is faster than the expected variation of physiologicalparameters, such as heart rate. The frame rate can also change in randomor pseudo-random ways, or through a set of non-stationary values, suchas three or more discrete, different frame rates. Further, a frequencypeak that remains stationary upon the change in frame rate is morelikely to be physiological. A stationary peak can be identified, and avital sign such as heart rate measured from this stationary peak. Aflowchart illustrating this method is shown in FIG. 17B. In anembodiment, the method includes adjusting the frame rate of the videosignal at 1711, and identifying peaks in the frequency domain thatchange or move with the adjusted frame rate at 1712. The method thenincludes filtering the source signal(s) to remove the identifiedfrequency at 1713. In an embodiment, after the noise frequency has beenidentified, the frame rate can be fixed, until a later time when it isvaried again to re-check for noise peaks.

The particular range of frame rates may depend on the capabilities ofthe camera hardware, and the light conditions. In an embodiment, theframe rate is varied from the highest well-exposed frame rate to lowerframe rate, in one or more steps. An example range is 10-25 frames persecond. In an embodiment, the period of time during which the frame rateis varied is longer than the expected period of the physiologicfrequency (such as heart rate). The analysis described in FIGS. 17A and17B can be done on each pixel signal individually, to remove identifiedfrequencies from each signal, or it can be done on one signal and thenthe identified frequencies can be removed from all pixel signals.

Optical Splitter

In another embodiment, an optical splitter is employed in order toobtain two light signals from a single camera. These two light signalsencompass the same field of view, monitoring the same subject, over thesame time period, but the two signals can be filtered differently tofacilitate physiological measurements. The two signals are synchronizedin time and field of view, and include the same noise components, so thesame de-noising operations can be used on both signals. The opticalsplitter is a simpler solution than two separate cameras, and providesmore information than a single camera.

An optical splitter 1810 according to an embodiment is shown in FIG. 18.The optical splitter 1810 is used to split a single light signal intotwo signals that pass through two different filters. The filters arechosen based on the physiologic signal that is to be measured. Forexample, for SpO2, the filters are chosen based on the extinctioncoefficients of hemoglobin. The two filters can pass visible andnon-visible light, respectively, such as red and infrared light, or twonarrow ranges of visible light. One filter may pass a narrow range ofred wavelengths, and the second filter may pass a narrow range of greenwavelengths, to mimic the red and infrared signals emitted bytraditional contact pulse oximetry emitters. Referring to FIG. 18, theoptical splitter includes an aperture 1812 that receives an incominglight signal 1800. The optical splitter includes a beam splitter 1814positioned behind the aperture, in the path of the incoming light. Thebeam splitter 1814 divides the incoming light signal 1800 into twosignals 1800A and 1800B. An example of a beam splitter is a dielectricmirror or a beam splitter cube, operating to split the incident lightinto two or more paths, in not necessarily equal proportions ofstrength. The separated light signal 1800B passes to a mirror 1816 thatre-directs the light signal into the camera. Each light signal passesthrough a respective filter 1820A, 1820B. In the example of pulseoximetry, the filter 1820A is designed to pass a narrow range of redwavelengths, while the filter 1820B passes a narrow range of greenwavelengths. The filtered light signals are received by respectivedetectors or light sensors 1822A, 1822B that register the resultingimages. The result is two time-varying image signals filtered forspecific wavelengths. Regions of interest are identified in the twosignals for the calculation of vital signs such as SpO2, heart rate, andrespiration rate, as described above. Absolute SpO2 can be calculatedvia a pre-calibrated look-up table, without the need for periodicre-calibration via oximeter spot check.

In another embodiment, additional splitters may be used to divide thelight into more than two beams, to pass through additional filterschosen for other physiologic parameters. For example, an additional beam1800N can be passed through a filter chosen for the measurement of totalhemoglobin. In another example, a filter is chosen forcarboxyhemoglobin, or for methemoglobin. In an embodiment, the filtersare arranged on a rotating wheel, so that they are rotated in and out ofthe path of the light 1800, 1800A, or 1800B, to filter the incominglight as needed for the measurement of the physiologic parameters. Thismechanical filter actuator can select appropriate filters to measuredifferent parameters from the patient at different times.

Data

Non-contact video-based monitoring methods have been employed in varioustest environments to confirm their utility. Some of that testing issummarized below. For example, FIGS. 19, 20, and 21 show video-basedmeasurements of heart rate, respiration rate, and SpO2 as compared toreference measurements, during a clinical study. During the study, thereference measurements were taken as follows: for heart rate and SpO2,from a contact-based pulse oximeter, and for respiration rate, from aventilator. A video camera was spaced apart from and oriented at theanimal subject, and video signals were captured through the course of anoxygen desaturation. The video signals were used to calculate heartrate, respiration rate, and SpO2, and these measurements were comparedto the reference measurements, as shown in FIGS. 19, 20, and 21. Thesefigures show good agreement between the video-based measurements and thereference measurements. Two separate regions of interest on the skinwere chosen, one for the determination of rates (RRvid and HRvid) andthe other for the determination of saturation (SvidO2).

FIG. 19 is a scatter plot of the video-based heart rate measurements onthe y-axis, against the reference heart rate measurements on the x-axis(both in beats per minute). The dotted line is a least squares fittedregression line. The expected 1:1 correspondence line is also shown, butis mostly hidden by the regression line, showing very good fit betweenthe two. Each desaturation episode is shaded separately.

FIG. 20 is a scatter plot of the video-based respiration ratemeasurements against the reference respiratory rate measurements fromthe ventilator (both in breaths per minute). The dotted line is a leastsquares fitted regression line. The expected 1:1 correspondence line isshown in solid black. The size of each circle corresponds to the numberof data points at that location; this visualization was required due tomany co-located data points in the plot.

FIG. 21 is a scatter plot of the video-based SpO2 measurements againstthe reference SpO2 measurements (both in %). The dotted line is a leastsquares fitted regression line. The expected 1:1 correspondence line isshown in solid black. Each desaturation episode is shaded separately.Changes in oxygen saturation were calculated using a ratio of ratiosderived from the red (R) and green (G) signals, where the two signalswere first normalised by dividing their cardiac pulse amplitudes by thesignal baseline values. As discussed above, using a standard RGB cameraonly allows for a relative saturation value to be determined from thisnormalised ratio of the amplitude of two of the signals. Hence thisrequired calibration against known values from the reference pulseoximeter to provide an absolute value of SvidO2.

The systems and methods described here may be provided in the form oftangible and non-transitory machine-readable medium or media (such as ahard disk drive, hardware memory, etc.) having instructions recordedthereon for execution by a processor or computer. The set ofinstructions may include various commands that instruct the computer orprocessor to perform specific operations such as the methods andprocesses of the various embodiments described here. The set ofinstructions may be in the form of a software program or application.The computer storage media may include volatile and non-volatile media,and removable and non-removable media, for storage of information suchas computer-readable instructions, data structures, program modules orother data. The computer storage media may include, but are not limitedto, RAM, ROM, EPROM, EEPROM, flash memory or other solid state memorytechnology, CD-ROM, DVD, or other optical storage, magnetic diskstorage, or any other hardware medium which may be used to store desiredinformation and that may be accessed by components of the system.Components of the system may communicate with each other via wired orwireless communication. The components may be separate from each other,or various combinations of components may be integrated together into amedical monitor or processor, or contained within a workstation withstandard computer hardware (for example, processors, circuitry, logiccircuits, memory, and the like). The system may include processingdevices such as microprocessors, microcontrollers, integrated circuits,control units, storage media, and other hardware.

Although the present invention has been described and illustrated inrespect to exemplary embodiments, it is to be understood that it is notto be so limited, since changes and modifications may be made thereinwhich are within the full intended scope of this invention ashereinafter claimed.

What is claimed is:
 1. A video-based method of measuring a patient'svital sign, comprising: receiving a video signal from a video camera,the video signal having a field of view that includes exposed skin of apatient; extracting from the video signal a time-varying color signalfor each of a plurality of regions, each region including exposed skinof the patient; identifying a frequency content of each color signal;selecting two or more non-adjacent, non-overlapping regions that have ashared frequency content comprising a modulation at a shared frequency;combining the color signals of the selected regions; measuring a vitalsign from the combined color signal; and outputting the vital sign forfurther processing or display.
 2. The method of claim 1, whereincombining comprises averaging the color signals from the selectedregions.
 3. The method of claim 2, wherein the selected regions havedifferent sizes.
 4. The method of claim 1, wherein the vital signcomprises heart rate.
 5. The method of claim 4, wherein measuring theheart rate comprises accumulating frequency peaks from the combinedsignal, selecting a median frequency, and updating a running averageheart rate from the selected frequency.
 6. The method of claim 3,further comprising updating the selected regions by adding or removingregions from the selected regions based on the frequency content of eachregion.
 7. The method of claim 1, further comprising calculating astatistic of the combined color signal, wherein the statistic comprisesan amplitude, a variability, a skew, or a signal to noise ratio.
 8. Themethod of claim 7, wherein combining the color signals comprisesapplying a weight to each color signal that is being combined, theweight being based on the statistic.
 9. The method of claim 7, furthercomprising determining a weight for the vital sign, the weight beingbased on the statistic, and adding the vital sign to a running averagebased on the weight.
 10. The method of claim 2, wherein selecting thetwo or more non-adjacent, non-overlapping regions further comprisesselecting regions that satisfy a quality criterion, and wherein thequality criterion comprises a signal to noise ratio that satisfies athreshold, or a skew value that satisfies a threshold.
 11. The method ofclaim 10, further comprising removing a color signal from the combinedcolor signal upon failure of the removed color signal to satisfy thecriterion.
 12. The method of claim 1, wherein extracting thetime-varying color signal comprises extracting two time-varying colorsignals for each region, and wherein measuring the vital sign comprisesmeasuring oxygen saturation from the two time-varying color signals. 13.A camera-based method of measuring a patient's vital sign, comprising:receiving an image signal from a camera having a field of viewencompassing exposed skin of a patient, the image signal comprising asequence of image frames over time; within the field of view,identifying a plurality of regions; for each region of the plurality,extracting an intensity signal comprising a time-varying light intensitydetected in the region; selecting regions whose intensity signalscomprise a modulation at a shared frequency, wherein the selectedregions are non-adjacent and non-overlapping in the image frame;combining the intensity signals of the selected regions to produce acombined intensity signal; and measuring heart rate of the patient fromthe combined intensity signal.
 14. The method of claim 13, wherein themeasured vital sign is a first measured vital sign, and wherein themethod further comprises: selecting a second set of regions whoseintensity signals comprise a second modulation at a second, different,shared frequency, wherein the second set of selected regions arenon-adjacent in the image frame; combining the intensity signals of thesecond set of selected regions to produce a second combined intensitysignal; and measuring respiration rate of the patient from the secondcombined intensity signal.
 15. The method of claim 14, furthercomprising displaying the image signal on a display, highlighting on thedisplay the selected regions for the first vital sign, and highlightingon the display the second, different set of selected regions for thesecond vital sign.
 16. The method of claim 13, wherein the modulation atthe shared frequency is above an amplitude threshold.
 17. The method ofclaim 13, wherein extracting the intensity signal for each regioncomprises extracting a red intensity signal comprising a time-varyingaverage of red values in the region, a green intensity signal comprisinga time-varying average of green values in the region, and a blueintensity signal comprising a time-varying average of blue values in theregion.
 18. A camera-based method of measuring a patient's vital sign,comprising: receiving an image signal from a camera having a field ofview encompassing exposed skin of a patient; receiving a user inputidentifying a region in the image signal; measuring a first vital signof the patient, comprising: from the region, selecting a first set ofpixels that exhibit a first modulation at a first shared frequency,wherein the first selected pixels are non-contiguous andnon-overlapping; and measuring the first vital sign of the patient fromthe selected first pixels; and outputting the first measured vital signfor further processing or display.
 19. The method of claim 18, furthercomprising: measuring and outputting a second vital sign of the patient,comprising: from the region, selecting a second set of pixels thatexhibit a second modulation at a second shared frequency different fromthe first shared frequency, wherein the second selected pixels arenon-contiguous; measuring the second vital sign of the patient from theselected second pixels.
 20. The method of claim 19, wherein the firstvital sign comprises heart rate and the second vital sign comprisesrespiration rate.
 21. The method of claim 18, further comprisingreceiving a second user input moving the region to a new location, andsubsequently measuring the first vital sign from the new location. 22.The method of claim 18, further comprising displaying a number derivedfrom the first measured vital sign, and triggering an alarm based on thedisplayed number and stored alarm conditions, wherein the stored alarmconditions omit a sensor-off or sensor-disconnect alarm.